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Abstract 

Developing a safe and reliable space vehicle requires good design and good 
manufacturing, or in other words “design it right and build it right”. A great 
design can be hard to build or manufacture mainly due to difficulties related 
to quality. Specifically, process control can be a challenge. As a result, the 
system suffers from low quality which leads to low reliability and high system 
risk. The Space Shuttle has experienced some of those cases, but has 
overcome these difficulties through extensive redesign efforts and process 
enhancements. One example is the design of the hot gas temperature sensor 
on the Space Shuttle Main Engine (SSME), which resulted in failure of the 
sensor in flight and led to a redesign of the sensor. The most recent example 
is the Space Shuttle External Tank (ET) Thermal Protection System (TPS) 
reliability issues that contributed to the Columbia accident. As a result, 
extensive redesign and process enhancement activities have been performed 
over the last two years to minimize the sensitivities and difficulties of the 
manual TPS application process. 

This paper discusses the importance of quality in system design, and the 



relationship between quality, reliability, and system safety for space vehicles. 
It uses examples from the Space Shuttle System with an emphasis on the ET 
TPS experience. It also discusses the redesign and process enhancement 
activities and shows how process control has improved TPS reliability and 
overall safety of the Space Shuttle vehicle in preparation for Return to Flight. 
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ABSTRACT 

Developing a safe and reliable space vehicle requires 
good design and good manufacturing, or in other words 
“design it right and build it right”. A great design can 
be hard to build or manufacture mainly due to 
difficulties related to quality. Specifically, process 
control can be a challenge. As a result, the system 
suffers from low quality which leads to low reliability 
and high system risk. The Space Shuttle has 
experienced some of those cases, but has overcome 
these difficulties through extensive redesign efforts and 
process enhancements. The most recent example is the 
Space Shuttle External Tank (ET) Thermal Protection 
System (TPS) reliability issues that contributed to the 
Columbia accident. As a result, extensive redesign and 
process enhancement activities have been performed 
over the last two years to minimize the sensitivities and 
difficulties of the manual TPS application process. 

This paper discusses the importance of quality in system 
design, and the relationship between quality, reliability, 
and system safety for space. 

1.0 BACKGROUND 

In the past, space vehicle designers focused more on 
performance and less on other system parameters. 

Reliability and safety was covered by designing for high 
safety factors. Safety factors are good if processes are in 
control and engineering analyses are bounding. 
However, past experience has shown that even for the 
best design, engineering analyses are not bounding in 
cases of excessive process variability and lack of 
process control. 

Developing a safe and reliable space vehicle requires 
good design and good manufacturing, or in other words 
“design it right and build it right”. Inadequate process 
control could result in low quality which leads to low 
reliability and high system risk. 

The difficulties and sensitivities of the Space Shuttle 
External Tank (ET) Thermal Protection System (TPS) 
manual spray process is a good demonstration of the 
impact of process control on component reliability and 
system risk. The TPS is a foam type material applied to 


the ET to maintain cryogenic propellant quality, 
minimize ice/frost formation, and protect the structure 
from ascent, plume, and re-entry heating. 

The ET main TPS components are shown in Fig. 1. ET 
main TPS components are applied by automated and/or 
manual processes. 



Fig. 1. ET Main Thermal Protection System 

As a result of the Columbia accident, some manually 
applied components of the TPS were 
enhanced/redesigned to reduce defects. A type of defect 
of main concern, which was the focus of the RTF 
activities, was the presence of voids within the TPS 
foam. Fig. 2 shows the enhanced/redesigned manually 
applied TPS components. 


Longeron Closeout 



Fig. 2. Enhanced/Redesigned ET Parts 

2.0 ET RETURN TO FLIGHT (RTF) LESSONS 
LEARNED 

The following sections discuss the lessons learned from 
ET return to flight (RTF) with regard to process control 



and its impact on TPS reliability and Space Shuttle risk. 
Section 2.1 addresses the relationship between process 
control, reliability and system risk. The rest of the 
sections address specific experiences from ET RTF. 

2.1 RELATIONSHIP BETWEEN PROCESS 
CONTROL, RELIABILITY, AND SYSTEM RISK 

Quality engineering, and more specifically process 
control, is the most important factor in reducing the 
Space Shuttle system risk. Good process control for ET 
TPS translate to lower number and smaller foam defect 
sizes in the TPS foam or more material capability. 
Lower defect numbers and smaller foam defect sizes 
translate to lower divot numbers and smaller divot sizes 
released in flight that could hit the Orbiter and cause a 
Space Shuttle catastrophic failure. In other words, 
higher TPS material capability means better TPS 
reliability and lower Shuttle risk. Fig. 3 shows the 
relationship between quality, reliability, and system 
risk. The following paragraphs discuss this relationship 
as applied to ET TPS foam. 



Fig. 3. Relationship between Process Control, 
Reliability, and System Risk 

We more often talk about process control in terms of 
Statistical Process Control (SPC). The scope of process 
control is much broader than SPC. Fig. 4 depicts the 
major elements of ET TPS process control or integrated 
process control (IPC). As shown in the figure, ET TPS 
IPC involved SPC, TPS application process control, 
manufacturing material control, contamination control, 
supplier process control, process change verification 
control, process monitoring, training and operator 
certification, and configuration management control. 

IPC was critical in ensuring consistent processes were 
employed for every part of the ET TPS. The focus of 
the ET project has been on SPC, standardization of 
spray techniques, early detection of changes in 
materials, comprehensive technician, operator and QC 
training, video review, process parameter data 
recording, and Quality Control (QC) inspection. The ET 
TPS SPC activity involved identification of process 
factors that affect the product quality, determination of 
the relative magnitude of the factors and the factors’ 
numerical sensitivity, and monitoring of the process 
critical factors. 
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Fig. 4. ET TPS Integrated Process Control 

The output of ET process control has been the most 
critical input to TPS reliability. As shown in Fig. 5, 
TPS reliability is defined in terms of TPS capability and 
system operating environment. TPS capability is 
defined in terms of material properties, process 
uniformity, and process capability. Process uniformity 
and process capability are characteristic of process 
defect frequency and size. For a manually sprayed TPS, 
the process uniformity and process capability are mainly 
driven by process control. In other words, ET TPS 
reliability is mainly driven by TPS process control. 

The output of TPS reliability is a set of probability 
distributions of TPS divot frequencies and divot sizes 
which are derived from the process defects when these 
defects are subjected to the flight operating 
environment. The TPS reliability output is a critical 
input to the Shuttle system risk assessment. 


Performance Capability 



TPS Operating jpg Material Properties 

Environment anc | process Capability 

Fig. 5. TPS Reliability 


TPS failure impact on Shuttle risk was evaluated using a 
probabilistic physics based engineering approach [1] 
[2]. Traditional Probabilistic Risk Assessment (PRA) 
involves all the scenarios that impact system risk [3]. 

The output of a PRA is an uncertainty distribution on 
system risk. The TPS probabilistic physics based 
engineering risk assessment focused on the impact of a 
failure mode on the system risk. The output was a point 
estimate of the risk. Confidence was heavily dependent 
on the level of conservatism of the engineering data and 
engineering assumptions. 





Sections 2.2 and 2.3 address the characterization and 
evaluation for both redesigned/enhanced and non 
redesigned (Use-As-Is) TPS. Section 2.4 describes the 
process that the Shuttle program and ET project used to 
assess the TPS reliability and system risk using the 
information and data characterization generated by the 
effort described in sections 2.2 and 2.3. 

2.2 Evaluation and Characterization of Redesigned ET 
TPS 

The following section discusses the approach used for 
improvements and evaluations related to manually 
applied TPS components. 

Manually applied ET TPS components were improved 
in two different aspects; a redesign of the TPS 
component, and an enhancement of the manual TPS 
application process specific to that component. 

ET TPS component redesign addresses the relationship 
between substrate geometry and defect formation. For 
example, the complexity of the underlying substrate was 
reduced, which corresponds to a reduction in the 
number and size of defects induced by complex 
substrates. 

Enhancement of the manual TPS application process 
included considerations for reduced operator to operator 
variability. For example, the sequence of operations 
were better organized and well defined with emphasis 
on operator training and certification specific to an ET 
TPS component. This allowed for a more consistent 
application process. 

Verification and validation testing of each TPS 
component redesign was performed, which provided 
sufficient data to evaluate and characterize the process 
variability and process capability. Process readiness 
was also evaluated using pre-control charts [4] . 
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Fig. 6. Example: Bipod Final Closeout Average Density 
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Fig. 7. Example: Bipod Closeout Slot Void Sizes 
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Fig. 8. Example: Bipod Closeout Cylinder Void Sizes 


Statistical evaluation of the data showed that significant 
improvements were made in process uniformity and 
process capability for material properties for the 
enhanced/redesigned ET TPS components. Significant 
reduction was detected in the coefficient of variation 
(COV) of the process critical output parameters (e.g. 
density, plug pull, voids, etc.). Fig. 6 shows an example 
of the Bipod Closeout redesign average density 
distribution. Furthermore, there was a significant 
reduction in the frequency and size of defects for the 
enhanced/redesigned ET TPS components. However, 
void characterization was still difficult because of 
limitation of the data and lack of good definition of the 
right tail of the data distribution [5]. Fig. 7, and Fig. 8, 
shows examples of the defect size distribution for the 
Bipod Closeout. 

For certification, a max expected void size was derived 
based on statistics and engineering analysis of the 
redesigned ET TPS component and then compared to an 
engineering limit derived from test data. 

2.3 Evaluation and Characterization of Use-As-Is TPS 


The following section discusses the evaluation of Use- 
As-Is TPS components. 


Process variability for Use-As-Is Foam was evaluated 
after the fact, without complete information about 
process variation and controls. For example, the natural 
variation of the process was not well understood, and 
the relationship between process control variables and 
defects was not known. 

The dissection data collected after the Columbia 
accident showed excessive variability (Coefficient of 
variation is greater than 100%) for process defect sizes 
and frequency. Within tank defect variability was high, 
and tank to tank defect variability could not be fully 
characterized due to limited data. Defect/void 
characterization was difficult and statistics derived had 
a high level of uncertainty. There was also a lack of 
random samples of sufficient size to empirically select a 
distribution for characterization. Furthermore, there 
was no engineering rationale to pick a specific 
distribution. Finally, there were very limited data to 
characterize the right tail of the distribution [6] . 

As a result of the above process control unknowns and 
data limitations, statistics was used only as supporting 
data for engineering evaluation and analysis. 
Additionally, engineering factors were used in the 
derivation of certification limits as a penalty to 
compensate for the lack of complete understanding of 
process controls and the statistical limitations of the 
data. 

2.4 Shuttle ET TPS Risk Assessment 

As mentioned earlier in the paper, the impact of TPS 
failure on Shuttle risk was evaluated using a physics 
based probabilistic engineering simulation approach. 
As shown in Fig. 9, the main input to the simulation 
model was the ET TPS void distributions derived from 
the dissection data of the ET components under 
consideration. The void distributions were then used in 
a fracture mechanics model to generate divots. The 
divots generated were then transported to evaluate the 
damage impact on the orbiter. The output of the model 
was the probability of Orbiter damage exceeding a 
specified tolerance limit set for the Orbiter. It is 
important to note that the void distributions, which 
included both the sizes of voids and the frequency of 
voids, represented the output of the ET TPS manual 
process which is basically driven by process control. 

The risk assessment model, although limited in scope, 
was very critical in understanding and communicating 
the risk of the ET TPS in flight. 
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Fig. 9. Shuttle ET TPS Risk Assessment Approach 
3.0 CONCLUSION 

Lessons learned from ET RTF experience demonstrates 
that a minor problem in process control could lead to a 
major problem at the system level which could 
significantly impact system risk. Consequently, good 
process control is essential in achieving high component 
reliability and low system risk. Manufacturing and 
process control should be considered up front in the 
design phase. Component designers of future launch 
vehicles should, consider manufacturability as well as 
the feasibility of good process control in the design 
selection process. To ensure consideration of process 
control throughout the program, an integrated process 
control plan should be developed upfront, and 
implemented throughout the different phases of the 
program. 
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Background 


• In the past, space vehicle designers focused on 
performance and less on other system parameters 

• Reliability and safety was covered by designing for high 
safety factors 

• Safety factors are good if processes are in control and 
engineering analyses are bounding. 

• Past experience has shown that even for the best 
design, engineering analyses are not bounding in cases 
of excessive process variability (lack of process 
control) 
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Background 


• Developing a safe and reliable space vehicle requires 
good design and good manufacturing, or in other words 
“design it right and build it right” 

• Inadequate process control could result in low quality 
which leads to low reliability and high system risk 

• The difficulties and sensitivities of the ET TPS manual 
spray process is a good demonstration of the impact of 
process control on component reliability and system 
risk 
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Background 



The Shuttle External Tank (ET) Thermal Protection 

System (TPS) 


The TPS is applied to the ET to maintain cryogenic 
propellant quality, minimize ice/frost formation, and 
protect the structure from ascent, plume, and re-entry 
heating 
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Background 

Thermal Protection System Overview 




1J 12 and L02 Intertank Flange Closeout 


L02 PAL Ramp 


Longeron Closeout 

LH2 Tank Dome / v L02 Feedline 


Aft Interfaces / Cable Tray 
Covers/F airings 


L02 Feedline Bellows 
LH2 Tank 


Bipod Closeouts 

Intertank Acreage 


Aft Struts 


LH2 PAL Ramp 


GH2 Pressline Fairing 


L02 Tank Ogive / 
Barrel 


Nose Cone 


F. Safie 


6 



Background 

Components and Process Enhanced Foam 
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Shuttle External Tank (ET) Return to Flight 

(RTF) Lessons learned 



Relationship Between process Control, Reliability, and System Risk 
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Shuttle External Tank (ET) Return to Flight 

(RTF) Lessons learned 
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Shuttle External Tank (ET) Return to Flight 

(RTF) Lessons learned 

ET TPS Integrated Process Control 



* Integrated process control (IPC) is critical to ensure consistent 
processes are employed for every part. Major activities in IPC 
for ET foam are: 

-Identification and control of critical processing variables 
-Standardization of spray techniques 
-Early detection of changes in materials and processes 
-Comprehensive technician, operator and QC training 
-Video review 

-Process parameter data recording 
-QC inspection 
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Shuttle External Tank (ET) Return to Flight 

(RTF) Lessons learned 


k 
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Statistical Process Control 

• Characterize the process to identify which process factors affect 
the product, the relative magnitude of the factors and the factors’ 
numerical sensitivity. 

• Determine critical process factors for process monitoring. The 
critical process factors and sources of variability is determined 
using tools such as Design of Experiments (DOE), regression 
analysis, etc. 

• Establish requirements and guideline for detection and adjustment 
of out of control processes. The guidelines shall be based on 
process characterization and experience 

• Define a sampling plan for product acceptance 

• Maintain an electronic record of process performance 
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Shuttle External Tank (ET) Return to Flight 

(RTF) Lessons learned 
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Statistical Process Control 


- A quality Engineering issue 

- Statistical in nature 

- TPS statistical data is extensive 

- The output of process control is a uniform and capable process 

- A critical input to the TPS reliability 
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Shuttle External Tank (ET) Return to Flight 

(RTF) Lessons learned 
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Reliability 


- Probabilistic in nature 

- It deals with capability (material properties and process 
capability) versus performance (system operating environment) 

- The output is a probability distribution 

- It is a critical input to the probabilistic system engineering and 
system risk assessment 
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Shuttle External Tank (ET) Return to Flight 

(RTF) Lessons learned 
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Shuttle External Tank (ET) Return to Flight 

(RTF) Lessons learned 

Evaluation and characterization of redesigned components and process 

enhanced foam 

* Approach for redesigned components and process enhanced foam 

- Improve process/design 

- Conduct verification and validation testing sufficient enough to 
understand and characterize the process variability and process 
capability 

- Evaluate process pre-control charts for process readiness 

- Evaluate process capability for meeting the specification 

- Evaluate process control for process uniformity 

- Statistically characterize process output for certification 
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Example 

Bipod Closeout Cylinder Voids 
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Shuttle External Tank (ET) Return to Flight 

(RTF) Lessons learned 

Evaluation and characterization of redesigned components and 

process enhanced foam 

• Statistical evaluation of the data for the enhanced/redesigned ET TPS 
showed: 

- Significant improvement in process uniformity and process capability 
for material properties 

- Significant reduction in frequency and size of defects 

- Significant reduction in the coefficient of variation (COV) of the process 
critical output parameters (e.g. density, plug pull, voids, etc.) 

- Better characterization of material properties. 

- Void characterization was still difficult because of limitation of the data 
and lack of good definition of the right tail of the data distribution 

• For Certification: 

- A max expected void size was derived based on statistics and 
engineering analysis 

- The max expected void size derived was compared to an engineering 
limit derived from test data 
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Shuttle External Tank (ET) Return to Flight 

(RTF) Lessons learned 

Evaluation and characterization of Use-as-is foam 

• Process variability was evaluated after the fact 

• Dissection data collected after the Columbia accident showed 
excessive variability (Coefficient of variation is greater than 100%) 

• Within tank variability was high, and tank to tank variability could 
not be fully characterized 

• Defect/void characterization was difficult and statistics derived had 
high level of uncertainty 

- There was a lack of random samples of sufficient size to 
empirically select a distribution 

- Very limited data to characterize the right tail of the distribution 

- There is no engineering rationale to pick a specific distribution 
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Shuttle External Tank (ET) Return to Flight 

(RTF) Lessons learned 



Evaluation and characterization of Use-as-is foam 


• The natural variation of the process was not well understood 

• Process controls related to manually-sprayed foam were related to 
environmental parameters. The relationship between process 
control variables and defects is not known 

• For certification: 

- A max expected void size was derived based on statistics and 
engineering analysis 

- The max expected void size derived was compared to an 
engineering limit derived from test data 
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Risk Assessment Using Probabilistic 
Engineering Approach 


k 

K 



- Traditional Probabilistic Risk Assessment (PRA) 

• Involves all the scenarios (multiple events) that impact system 
risk 

•The output is an uncertainty ( or confidence) distribution on 
system risk (e.g. the risk is less than 1 in 1000 with 50% 
confidence) 

- Probabilistic Engineering Assessment focuses on the impact of 
a failure mode on the system risk 

• The output, in general, is a point estimate of the risk (e.g. the 
risk is 1 in 1000). 

• Confidence is heavily dependent on the level of conservatism 
of the engineering data and engineering assumptions 

• Results could be used as an input to a basic event or a single 
scenario in the traditional PRA process. 
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Conclusions 




• Process control is a critical factor for achieving high 
reliability and low system risk 

• Component designers of future launch vehicle should, 
consider manufacturability and the feasibility of good 
process control in the design selection process 

• An integrated process control plan should be developed 
upfront, and implemented throughout the different phases 
of the program 
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